Summary

Two independent, cautious changes to how the crawler handles target-domain rate limiting (429/403/503). Motivated by a 10k-task load test where 44% of tasks hard-failed, almost entirely on HTTP 429 — DB and infra were healthy, so the dominant limiter on successful throughput is target-domain throttling, not infrastructure.

1. Raise blocking retries (deferred, not failed)

• MaxBlockingRetries was hardcoded to 3, so a rate-limited task burned all attempts in <1s (backoff 100/200/400ms) and hard-failed while the domain was still throttling.
• Now reads GNH_RATE_LIMIT_MAX_RETRIES (previously documented in ENV_VARS.md but never wired) with a raised default of 8, so a persistently-throttling domain is paced-and-deferred across the widening per-domain delay (up to the 60s cap) rather than counted as a failure.

2. Pace every job start (no first-wave burst)

• The pacer warm-up delay was only applied to never-crawled domains. A domain with a learned (or zero) adaptive delay seeded 0 → no per-domain cap → the full per-job concurrency burst out on the first dispatch wave and instantly triggered a 429 storm.
• Job starts are now floored to a minimum seed delay (GNH_PACER_START_FLOOR_DELAY_MS, default 500ms, 0 restores prior behaviour). A previously-crawled domain now starts at ~3 concurrent requests and widens on success, instead of bursting. A never-crawled domain keeps the cautious 2000ms warm-up (~1 worker). A domain with a higher learned delay is unchanged.
• Seed decision extracted into seedAdaptiveDelayMS(...) and unit-tested.

Resulting job-start ramp

The cap then widens to full job concurrency via the existing success step-down.

Reviewer notes

• Both env vars default to the new behaviour; set GNH_PACER_START_FLOOR_DELAY_MS=0 to revert Part 2.
• The per-domain spacing logic itself is unchanged (first 429 → 500ms, +500ms per further 429, up to 60s).
• Known pre-existing sharp edge (not introduced here): when the adaptive delay reaches 0 the cap is removed entirely, so the final ramp step jumps from ~3 to full rather than incrementing. Candidate for a follow-up.
• Load-test evidence is synthetic (hammering domains); worth confirming against real customer traffic before further tuning.

Test plan

• go build ./...
• go test ./internal/jobs/ ./internal/broker/
• New unit tests for seedAdaptiveDelayMS (warm-up, floor, disabled-floor cases)
• gosec clean; pre-commit security gates pass

^{Need help on this PR? Tag @codesmith with what you need. Autofix is disabled.}

Summary by CodeRabbit

• Improvements

  • Rate-limited task failures now retry up to 8 times (was 3); max retries configurable via environment.
  • Jobs use a paced start to avoid initial concurrency bursts; start-floor delay configurable (default 500ms) and can be disabled.

• Documentation

  • Updated runtime config docs with new env vars and defaults (max retries, pacer warmup delay default 2000ms, start-floor default 500ms).

• Tests

  • Added unit tests covering pacing and retry configuration and defaults.